For most visual tracking applications, measurement data are uncertain and sometimes missing: images are taken with noise and distortion, while occlusions can render part of the object-of-interest unobservable. Uncertainty can be globally uniform; but in most real-world scenarios, it is heteroscedastic in nature, i.e., both anisotropic and inhomogeneous. A good example is the echocardiogram (ultrasound heart data). Ultrasound is prone to reflection artifacts, e.g., specular reflectors, such as those that come from membranes. Because of the single “view direction”, the perpendicular surface of a specular structure produces strong echoes, but tilted or “off-axis” surfaces may produce weak echoes, or no echoes at all (acoustic “drop out”). For an echocardiogram, the drop-out can occur at the area of the heart where the tissue surface is parallel to the ultrasound beam.
Due to its availability, relative low cost, and noninvasiveness, cardiac ultrasound images are widely used for assessing cardiac functions. In particular, the analysis of ventricle motion is an efficient way to evaluate the degree of ischemia and infarction. Segmentation or detection of the endocardium wall is the first step towards quantification of elasticity and contractibility of the left ventricle. Examples of some existing methods include pixel-based segmentation/clustering approaches (e.g., Color Kinesis), variants of optical flow, deformable templates and Markov random process/fields, and active contours/snakes. Some methods are employed in 2-Dimensional, 3-Dimensional or 4-Dimensional (3D+time) space.
However, most existing segmentation or detection methods do not attempt to recover accurate regional motions of the endocardial wall, and in most cases, motion components along the wall are ignored. This simplified treatment is also employed by contour trackers that search only along the normals of the current contour. This is not suitable for regional wall abnormality detection, because regional motion of an abnormal left ventricle is likely to be off the normal of the contour, not to mention that global motion, such as translation or rotation (due to the sonographer's hand motion or respiratory motion the patient), causes off-normal local motion on the contour as well. It is desirable to track the global shape of endocardial wall as well as its local motion, for the detection of regional wall motion abnormalities. This information can be used for further diagnosis of ischemia and infarction.
In general, covariances can be assigned to image features or flow estimates that reflect underlying heteroscedastic noise. When the data is clean with a low overall noise level, the heteroscedastic nature may be ignorable, and a global uncertainty can be substituted for the local estimates. However, for very noisy inputs, especially those with spatially varying structural noise, the information encoded in the local covariance matrix becomes critical in ensuring reliable and robust inference of objects or underlying image structures.
It is a common practice to impose model constraints in a tracking framework. Examples include simple models such as blobs or parameterized ellipses, and complex models such as discriminative templates. In most practical cases, a subspace model is suitable for shape tracking, since the number of modes capturing the major shape variations is limited and usually much smaller than the original number of feature components used to describe the shape. Furthermore, a Principal Component Analysis (PCA)-based eigenshape subspace can capture arbitrarily complicated shape variations, which, in the original space, even with a very simple parametric model, are highly nonlinear.
If a measurement vector is affected by heteroscedastic noise, an orthogonal projection into the constraining subspace is not only unjustified, but also very damaging in terms of information loss. It can only be justified for the special case when the noise is both isotropic and homogeneous.
However, most existing work on subspace-constrained tracking does not take into account the heteroscedastic noise in the measurements. In the “Point Distribution Model” or “Active Shape Model”, a PCA-based subspace shape model is derived based on training shapes with landmark point correspondence. The resulting subspace of eigenshapes captures the most significant variations in the training data set. At detection time, a model is perturbed to create synthetic images for matching against the testing image at a candidate location. However, the measurement noise was not modeled in this process.
Even when heteroscedastic noise characteristics are available, they were typically disregarded during the subspace model fitting. For example, in one known approach where full covariance matrix was captured in the measurements, a rather ad hoc thresholding is applied so that the measurement mean is confined to a hyper-ellipsoid constraint defined by the model covariance. This operation is independent of the measurement noise.
Another known approach applies a two-step approach to impose a shape space constraint in a Kalman filtering framework. The shape space is a linearly transformed affine subspace or eigen-subspace. However, the projection into the shape space is orthogonal, without taking into account the heteroscedastic noise of the measurement. Therefore, this approach leads to information loss during the projection.
Another known approach uses a Gaussian distribution to adaptively model the appearance of the object of interest (face in their case), which is learned using the EM algorithm. As in the present invention, local uncertainty is captured in the covariance matrix. The difference is that the present invention specifically studies the subspace model constraints and the critical choice of intersection over projection when anisotropic uncertainty is present.
Another known approach uses a subspace constraint implicitly during the optical flow estimation, which also utilizes flow uncertainties. Although in a different framework for a different application, present invention recognizes that “more reliable flow-vectors will have more influence in the subspace projection process.”
Another known approach applies heteroscedastic regression for fitting ellipses and fundamental matrices. The fitting is achieved in the original space with parameterized models. In the present invention, parameterization of shape variations is avoided—it can be very complicated and highly nonlinear. Instead, the present invention builds subspace linear probabilistic models through, e.g., PCA, and obtain closed-form solutions on both the mean and covariance of the fitted data.
Robust model matching relying on M-estimators or RANSAC has also been applied to limit or eliminate the influence of data components that are outliers with respect to the model. Again, the locally (in space or time) varying uncertainties are not exploited in these frameworks.
Other related approaches include data imputation, the practice of “filling in” missing data with plausible values. Work in this area is rooted in statistics with broad applications toward speech recognition, medical image analysis, and social science, etc. However, the formulation of data imputation problems typically assumes 0-1 availability, i.e., a data component is either missing or available. There is a need for a unified framework for fusing subspace model constraints with information about the shape dynamic and the heteroscedastic nature of the measurement noise and about the shape dynamics.